***Predicting the GENRE Of The Movie¶
In [13]:
THRILLER_MOVIE
Out[13]:
In [15]:
COMEDY_MOVIE
Out[15]:
In [19]:
SuperNatural_Action_thriller
Out[19]:
In [20]:
Action_Romantic_Thriller
Out[20]:
In [22]:
Animated_movie
Out[22]:
In [24]:
Action_Adventure
Out[24]:
In [26]:
science_fiction_movie
Out[26]:
In [28]:
fantacy_movie
Out[28]:
In [29]:
Action_movie
Out[29]:
In [31]:
Romantic_movie
Out[31]:
* We have to predict the of the movie Genre based on title, tagline, original_title and overview fields.¶
* In general, we predict total collections, hit or flop, which might be a Regression And Classification Problem¶
* Here we have to predict the Genre of the movie which is very complicated because each record is having multiple outputs and target column is having multiple classes.¶
The outline is as follows:¶
1.The data is imported and prepared for analysis.¶
2.The features are analysed and appropriate choices made to improve the fit.¶
3.Genre column is changed.¶
4.converting words to vectors by applying vectorization method.¶
5.Test set result of final model.¶
6.A short discussion on future work.¶
In [0]:
import numpy as np
import pandas as pd
pd.set_option('display.max_columns' , 100)
pd.set_option('display.max_rows' , 100)
In [0]:
import os
os.chdir("/content/drive/My Drive/movie data set")
In [0]:
data=pd.read_csv("moviesdatase.csv")
* We are taking the following required columns only for analysis.¶
In [0]:
final_data2=pd.read_csv("moviesdatase.csv",usecols=["tagline","title","original_title","overview","genres"])
In [0]:
final_data2.head(5)
Out[0]:
* We are having about 45,000 records when we are using it directly we had issues with RAM as data was huge because we have to countvectorize all the words for all the plot overview resulting it to crash. So I have choosen to use only 20000 records after trying with 30000 and 25000 records respectively.¶
* This is because in Movie Summary we will be having so many so words and we for model building we need to convert them into numeric of vector form then we will face the problem of dimensionality and more over computational problem is also there.¶
In [0]:
final_data=final_data2.iloc[range(20000),:]
In [0]:
def summary(final_data):
print('Shape of data :' ,final_data.shape )
return(pd.DataFrame({"Datatype":final_data.dtypes ,
"NAs":final_data.isnull().sum() ,
"Uniques":final_data.nunique(),
"levels": [final_data[x].unique() for x in final_data.columns]}))
In [0]:
summary(final_data)
Out[0]:
In [0]:
final_data.to_csv("moviesdatase.csv",index=False)
In [0]:
final_data["genres"][0]
Out[0]:
*We have to do some preprocessing steps for text data.¶
*Stop Words are those which do not have any individual meaning¶
* These are the words which are not considered by the search enegine also.¶
* In data pre-processing mainly in NLP we have 2 steps Stemming and Lemmatization. Stemming is cutting of Prefix or suffix Word some times it may not have any meaning also. In Lemmatization is also chopping of words and it will be having meaning also but computationally we need so much power so we use Stemming.¶
In [0]:
import json
import nltk
import re
import csv
import matplotlib.pyplot as plt
import seaborn as sns
from tqdm import tqdm
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
%matplotlib inline
pd.set_option('display.max_colwidth', 300)
In [0]:
import ast
from ast import literal_eval
* In genre column we are having keys and values in the form of a Dictionary so we have extracted only the values because we dont use keys, Keys are just the representation of keys.¶
In [0]:
final_data['modified_genres'] =final_data['genres'].apply(lambda x:[genre['name'] for genre in literal_eval(x)])
final_data.head()
Out[0]:
In [0]:
final_data.drop("genres",axis=1,inplace=True)
In [0]:
final_data.head()
Out[0]:
*Creating Y variables - Dealing with multi labels¶
*It seems that choosing genre data is a complicated process as some movies have multiple genres associated. In order for our model to train and predict from one genre per each observation, we duplicated movie data for each genre. So if a movie has two genres, we will have an observation for each genre¶
In [0]:
from itertools import combinations, permutations
In [0]:
final_data['pairs_genre'] = final_data.modified_genres.map(lambda x : list(combinations(x, 2)))
final_data['uni_genre'] = final_data.modified_genres.map(lambda x : list(combinations(x, 1)))
*Here we are creating a unique column, each record having multiple column will created as a seperate list.¶
In [0]:
melted_uni_genre = pd.DataFrame(pd.concat([pd.Series(row['original_title'], row['uni_genre']) for x,
row in final_data.iterrows()])).reset_index()
In [0]:
melted_uni_genre.columns = ['uni_genre', 'original_title']
*Here we are Creating a pair genre column where a pair of genres will be in the list.¶
In [0]:
melted_pairs_genre = pd.DataFrame(pd.concat([pd.Series(row['original_title'], row['pairs_genre']) for x,
row in final_data.iterrows()])).reset_index()
In [0]:
melted_pairs_genre.columns = ['pairs_genre', 'original_title']
In [0]:
melted_pairs_genre.head()
Out[0]:
In [0]:
melted_uni_genre.head()
Out[0]:
In [0]:
from collections import Counter
* Here we can get a list of how many times genre had appeared in the total record.¶
In [0]:
Counter(melted_uni_genre.uni_genre)
Out[0]:
* Here we are having a list how many times pair of genres have appeared in the list.¶
In [0]:
Counter(melted_pairs_genre.pairs_genre)
Out[0]:
In [0]:
final_data.columns
Out[0]:
* Here we have merged the final data with Data frame having Unique genres and pair of geners.¶
In [0]:
merged_mdb_uni = pd.merge(melted_uni_genre,final_data , how ='left', on = 'original_title');
In [0]:
merged_mdb_pair = pd.merge(melted_pairs_genre, final_data , how ='left', on = 'original_title')
In [0]:
merged_mdb_uni.shape, merged_mdb_pair.shape
Out[0]:
In [0]:
merged_mdb_uni.columns
Out[0]:
In [0]:
merged_mdb_pair.columns
Out[0]:
In [0]:
merged_mdb_uni.head(1)
Out[0]:
* Here we have converted "Unique Genre" column into categorical because we are applying label encoing to converting into numeric form.¶
In [0]:
merged_mdb_uni["uni_genre_x"]=merged_mdb_uni["uni_genre_x"].astype("str").astype("category")
Label Encoding for the Unique Genre column¶
In [0]:
#Encode
merged_mdb_uni2 = merged_mdb_uni
#Convert Y variable to numerical value prior to modelling
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(merged_mdb_uni2['uni_genre_x'])
merged_mdb_uni2['uni_genre_x'] = le.transform(merged_mdb_uni2['uni_genre_x'])
In [0]:
merged_mdb_uni2.head(1)
Out[0]:
* Count Vectorizer Just counts the words, how many times it had occured where as Tfidf vectorizer counts the number of time word occured and also tells us how significant the word is.¶
In [0]:
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
import nltk
nltk.download('stopwords')
nltk.download('punkt')
stemmer= PorterStemmer()
from nltk.corpus import stopwords
* Here we are Creating a function for the pr-processing steps mentioned above.¶
In [0]:
def process(doc):
import re
stop=stopwords.words('english')
doc=re.sub("[\d_]+", " ",doc)
doc=re.sub("[\W_]+"," ",doc)
x= word_tokenize(doc)
x=[i for i in x if len(i)>5]
stemmed=[stemmer.stem(word) for word in x]
clean=[word for word in stemmed if not word in stop]
joined= ' '.join(clean)
return joined
* Here we are converting 3 columns in to vector form "overview","Tagline","original_title" because these are the columns in sentence form.¶
In [0]:
final_data['overview'].fillna(value=np.nan, inplace=True)
print(final_data['overview'])
from sklearn.feature_extraction.text import CountVectorizer
#Overview
vectorizer = CountVectorizer(stop_words='english',decode_error='ignore',analyzer='word', min_df=4,preprocessor=process)
corpus = final_data['overview'].values.astype('U')
wordvec = vectorizer.fit_transform(corpus.ravel())
wordvec = wordvec.toarray()
words = vectorizer.get_feature_names()
print("number of words in plot summary: ", len(words))
overview_db = pd.DataFrame(wordvec,columns=words)
final_data['original_title'].fillna(value=np.nan, inplace=True)
print(final_data['original_title'])
#original_title
vectorizer = CountVectorizer(stop_words='english',decode_error='ignore',analyzer='word', min_df=4,preprocessor=process)
corpus = final_data['original_title'].values.astype('U')
wordvec = vectorizer.fit_transform(corpus.ravel())
wordvec = wordvec.toarray()
words = vectorizer.get_feature_names()
print("number of words in title: ", len(words))
original_title_db = pd.DataFrame(wordvec,columns=words)
final_data['tagline'].fillna(value=np.nan, inplace=True)
print(final_data['tagline'])
#original_title
vectorizer = CountVectorizer(stop_words='english',decode_error='ignore',analyzer='word', min_df=4,preprocessor=process)
corpus = final_data['tagline'].values.astype('U')
wordvec = vectorizer.fit_transform(corpus.ravel())
wordvec = wordvec.toarray()
words = vectorizer.get_feature_names()
print("number of words in tagline: ", len(words))
tagline_db = pd.DataFrame(wordvec,columns=words)
*Here we are Creating a Data Frame for all the 3 columns that are converted in to vector form¶
In [0]:
overview_db = pd.DataFrame(overview_db)
original_title_db = pd.DataFrame(original_title_db)
tagline_db=pd.DataFrame(tagline_db)
In [0]:
tagline_db.head()
Out[0]:
In [0]:
merged_mdb_uni3 = merged_mdb_uni2
In [0]:
merged_mdb_uni3.columns
Out[0]:
* We are dropping the columns which we have have already converted into vector form¶
In [0]:
merged_mdb_uni3 = merged_mdb_uni3.drop(['pairs_genre'], axis=1)
merged_mdb_uni3 = merged_mdb_uni3.drop(['overview'], axis=1)
merged_mdb_uni3 = merged_mdb_uni3.drop(['original_title'], axis=1)
In [0]:
merged_mdb_uni3 = merged_mdb_uni3.drop(['tagline'], axis=1)
In [0]:
merged_mdb_uni3 = merged_mdb_uni3.drop(['title'], axis=1)
In [0]:
merged_mdb_uni3.head()
Out[0]:
*We are the concatinating al the 3 vector converted columns to unique converted Genre column.¶
In [0]:
merged_mdb_uni3 = pd.concat([merged_mdb_uni3, overview_db], axis=1)
In [0]:
merged_mdb_uni3 = pd.concat([merged_mdb_uni3, original_title_db], axis=1)
In [0]:
merged_mdb_uni3 = pd.concat([merged_mdb_uni3, tagline_db], axis=1)
In [0]:
merged_mdb_uni3.head()
Out[0]:
In [0]:
dataset = merged_mdb_uni3
In [0]:
dataset = dataset.dropna(axis = 0)
In [0]:
dataset.drop("uni_genre_y",axis=1,inplace=True)
In [0]:
dataset.head()
Out[0]:
In [0]:
dataset["uni_genre_x"]=dataset["uni_genre_x"].astype("str").astype("category")
* We have dropped the main Genre column because we have kept numerically converted "unique_genre" column as output.¶
In [0]:
dataset.drop("modified_genres",axis=1,inplace=True)
In [0]:
dataset['uni_genre_x'].value_counts()
Out[0]:
In [0]:
from sklearn.model_selection import cross_validate
In [0]:
from sklearn import discriminant_analysis
from sklearn.decomposition import PCA
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression as LogReg
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from scipy.stats import mode
from sklearn import linear_model
from IPython.core.interactiveshell import InteractiveShell
from time import sleep
from collections import Counter
from itertools import combinations, permutations
#from urlparse import urljoin
In [0]:
dataset.head()
Out[0]:
MODEL BUILDING¶
In [0]:
from sklearn.model_selection import train_test_split
xtrain,xtest,ytrain,ytest = train_test_split(dataset.drop('uni_genre_x',axis=1), dataset['uni_genre_x'],test_size=0.2)
Logistic REgression¶
In [0]:
from sklearn.linear_model import LogisticRegression
lg = LogisticRegression(class_weight='balanced',max_iter =1000)
lg.fit(xtrain,ytrain)
print(lg.score(xtest,ytest))
In [0]:
lg.score(xtrain,ytrain)
Out[0]:
Random Forest¶
In [0]:
from sklearn.ensemble import RandomForestClassifier
In [0]:
lg2= RandomForestClassifier()
lg2.fit(xtrain,ytrain)
print(lg2.score(xtest,ytest))
In [0]:
lg2.score(xtrain,ytrain)
Out[0]:
In [0]:
from xgboost.sklearn import XGBClassifier
In [0]:
from sklearn.tree import DecisionTreeClassifier
Decision Trees¶
In [0]:
lg3= DecisionTreeClassifier(max_depth=7,class_weight='balanced')
lg3.fit(xtrain,ytrain)
print(lg3.score(xtest,ytest))
In [0]:
lg3.score(xtrain,ytrain)
Out[0]:
Recommendations¶
In [0]:
### *Logistic Regression and Random Forest lead to over-fitting